In a Linux or UNIX operating system (OS) everything that accepts or sends data to or from the operating system is considered to be a “device”. Common devices include terminals, hard disks, printers, CD drives, and modems. A rule of thumb is that anything used for input, storage, or handling of data in an OS is considered a device and needs a device entry so that it may be detected by the OS.
All devices communicate in one of two ways; either one character at a time, or as a complete set of data in a predefined block. Therefore, devices can be classified as being either “character mode” or “block mode”. Terminals, printers, and modems are character devices since they handle data one character at a time. Hard drives and tape drives use blocks to transmit data as large chunks of information so they are block mode devices. Some devices function in either mode using two different device drivers.
Each of these devices may also be SCSI devices. SCSI (small computer system interface) is a family of standards that connects components via a parallel bus for system-level interfacing between a computer and intelligent devices such as hard disks, floppy disks, CD ROMs, printers, scanners and other types of media devices or storage devices. SCSI can connect multiple devices to a single adaptor (e.g., a host bus adapter (HBA) or port) on a SCSI bus. SCSI devices transfer bits in parallel and can operate in either asynchronous or synchronous modes.
A device driver is a software module that is part of the kernel and that supports access to a peripheral device. A device driver is a collection of subroutines and data within a kernel (a kernel is the core software of an OS) that constitutes the software interface to an I/O device. In Linux, when a device needs to be accessed the Linux kernel passes an I/O request to the device driver, which handles the communication with the device. Special device files (hereafter “device files”) are used to access devices which are connected to a Linux system, devices such as printers, tape drives, terminals and disk drives used for file system storage. Device files are kept in what is called a /dev directory and are given device file names representative of their function, such as “/dev/fd0” for a floppy drive, for example. A device file may be type c (for “character” devices, devices that do not use a buffer cache) or type b (for “block” devices, which go through a buffer cache). In Linux, all disks are represented as block devices only.
Device numbers allow the operating system to differentiate between the devices. In a standard Linux implementation, each device is assigned a major number and a minor number. The major number identifies the driver, while the minor number identifies the instance of the device. For example, the device file name “/dev/fd0” identifies the first floppy drive on a system, with “fd” representing the major number, and “0” representing the minor number of the device file name. Before discussing major and minor numbers in greater detail, however, the Linux SCSI I/O Subsystem driver architecture and SCSI addressing in Linux is briefly explained.
Driver Architecture Overview.
FIG. 1 illustrates a driver architecture for the Linux SCSI I/O Subsystem. The Linux SCSI I/O subsystem (hereinafter SCSI subsystem) has a three (3) level driver architecture with the “upper” level layer 100 being closest to a user space/kernel space interface 110, while the “lower” level layer 150 is closest to hardware (e.g., physical SCSI device, not shown). The upper level drivers (ULDs) in upper level 100 are commonly known by a terse two letter abbreviation (e.g. “sd” for SCSI disk driver, “st” for SCSI tape driver, “sr” for SCSI CD-ROM driver; and “sg” for a SCSI generic driver). The names of the corresponding module drivers which, for historical reasons, sometimes differ from the built-in driver names are shown in brackets in FIG. 1.
The ULDs in upper level layer 100 maintain the kernel side of the Linux OS's user-kernel interface for the logical class of devices they represent (e.g. disks). They are also responsible for managing certain kernel and SCSI subsystem resources such as kernel memory and SCSI command structures. Applications in the user space (e.g., user daemons) access these ULDs by opening a device file (block or char) typically found in the /dev directory tree.
Accordingly, the upper level layer 100 supports the kernel/user interface. For example, in the case of sd (disk) drivers and sr (CD-ROM) drivers, the kernel/user interface is a block device interface, while for st (tape) drivers and sg (generic) drivers, the kernel/user interface is a character device interface. Devices can be classified as being either “character mode” or “block mode”. Terminals, printers, and modems are character devices since they handle data one character at a time. Hard drives and tape drives use blocks to transmit data as large chunks of information so they are block mode devices. Some devices function in either mode using two different device drivers.
The mid-level SCSI layer 125 is common to all operations. The mid-level SCSI layer 125 defines internal interfaces and provides common services to the upper and lower level drivers. IOCTLs (input/output control commands) provided by the mid-level SCSI 125 layer are available to file descriptors belonging to any of the four ULDs (sd, sr, st, sg device drivers). Any operation using the SCSI subsystem (e.g. reading a sector from a disk) involves one driver at each of the three levels, e.g. sd drivers at upper level 100, SCSI mid-level driver at SCSI mid level layer 125 and an HBA driver of lower level layer 150.
SCSI Subsystem Addressing.
Linux has a four level hierarchical addressing scheme for SCSI devices: (a) SCSI adapter number [host]; (b) channel number [bus]; (c) ID number [target]; and (d) LUN [lun]. The term “LUN” is a common SCSI abbreviation of Logical Unit Number. The LUN identifies the logical unit within a SCSI target device. A SCSI target device is a SCSI device containing logical units (LUs) that service commands from a SCSI initiator device in order to carry out a specified task for example. As is known, a logical unit may be defined as addressable blocks of storage created from one or more disks (also called spindles) contained within a SCSI device. A logical unit may provide a unique connection to an application program or another SCSI device. “Bus” is used herein in preference to “channel” in the description below.
The SCSI adapter number is typically an arbitrary numbering of adapter cards on internal I/O SCSI buses (e.g. PCI, ISA, etc.) of a computer (Linux host) running the Linux OS. Such adapters are commonly referred to as host bus adapters (HBAs). HBA numbers are issued by the Linux kernel in ascending order, starting with 0 and proceeding for N HBAs. Each HBA may control one of more SCSI buses.
Each SCSI bus can have multiple SCSI devices connected to it. In SCSI parlance, the HBA may be called the “initiator” (e.g., it may be an HBA of a SCSI initiator device) and takes up one SCSI ID number (typically the number “7”). The initiator talks to targets, commonly also known as SCSI target devices (e.g. disks), as briefly discussed above.
On SCSI parallel buses, the number of SCSI IDs are related to the width of the bus. For example, 8-bit buses (sometimes called “narrow” SCSI buses) may have 8 SCSI IDs, of which one ID is taken by the HBA, leaving 7 IDs for SCSI devices. Wide SCSI buses are 16 bits wide and can have a maximum of 15 SCSI devices (e.g., targets) attached. The SCSI-3 draft standard allows a large number of IDs to be present on a SCSI bus, (0-255) with each SCSI device adapted to contain multiple (LUNs). These multiple LUNs are typically used by sophisticated tape and CD-ROM units that support multiple media. Accordingly, Linux's flavor of SCSI addressing is a four level hierarchy: <scsi(_adapter_number), channel, id, lun>, which is most commonly known and denoted with <host, bus, target, lun> encoding, i.e., (h0b0t0l0). Thus, a path from a SCSI host to a SCSI device, or to a logical unit represented by a LUN on a SCSI device, can be determined from the (h, b, t, l) address of a logical unit.
Storage area networking is predicated on the replacement of parallel SCSI transport with networked storage SCSI devices and tape SCSI devices behind the server or Linux Host. The vast majority of storage area networks (SANs) use a Fibre Channel (FC) medium for the underlying network transport and, at the upper layer, move data to and from disks with serial SCSI protocol, as described in the current draft SCSI Primary Commands-3 (SPC-3) document. The latest version of the SPC-3 document, found at ftp://ftp.t10.org/t10/drafts/spc3/spc3r09.pdf, provides SCSI primary command standards that are designed to be used industry wide. The combination of a high-speed transport (FC) with native SCSI protocol results in an efficient means to deploy servers, disk arrays, and tape subsystems and frees all components from the constraints of parallel SCSI architecture.
The SCSI-3 device driver supplied by the HBA vendor is responsible for mapping FC storage resources to the bus, target, lun conventions required by Linux. Since FC addresses are self-configuring, the mapping between port addresses, (which may change) and the upper level SCSI device driver designations are maintained by the HBA and the HBA's device driver interface to Linux.
The driver architecture and addressing within the SCSI Subsystem having been introduced, device names and their structure are now explained. A device name can be thought of as gateway to a Linux kernel driver (e.g., ULD sd, sr, st, sg) that controls a device rather than a gateway directly to the device itself. Hence there can be multiple device names, some of which may offer slightly different characteristics, all mapping to the same actual device.
The device names of the various SCSI devices may be found within the /dev directory. Traditionally in Linux, SCSI devices have been identified by their major and minor device number, rather than by their SCSI bus addresses (e.g. SCSI target ID and LUN). Eight block major numbers are reserved for SCSI disks: 8, 65, 66, 67, 68, 69, 70 and 71. Each major number can accommodate 256 minor numbers which, in the case of SCSI disks, are subdivided as follows in the following Table 1:
TABLE 1Device Names for SCSI Disk Devices (sd) - Major No. 8[b,8,0]/dev/sda[b,8,1]/dev/sda1. . .[b,8,15]/dev/sda15[b,8,16]/dev/sdb[b,8,17]/dev/sdb1. . .[b,8,255]/dev/sdp15
In Table 1, the disk device names without a trailing digit refer to the whole disk (e.g. /dev/sda) while those with a trailing digit refer to one of the 15 allowable partitions within that disk. Linux supports up to 15 partitions per virtual whole disk, so the minor number serves to distinguish partitions form each other. Further, the disk device names without a trailing digit refer to the whole disk (e.g. /dev/sda) while those with a trailing digit refer to one of the 15 allowable partitions within the disk.
Table 2 illustrates naming conventions for the remaining seven SCSI disk block major numbers. The remaining 7 SCSI disk block major numbers (e.g., 65, 66, 67, 68, 69, 70 and 71 follow a similar pattern:
TABLE 2SCSI Disk Devices (sd) Names - Remaining Major Nos.[b,65,0]/dev/sdq[b,65,1]/dev/sdq1. . .[b,65,159]/dev/sdz15[b,65,160]/dev/sdaa[b,65,161]/dev/sdaa1. . .[b,65,255]/dev/sdaf15[b,66,0]/dev/sdag[b,66,1]/dev/sdag1. . .[b,66,255]/dev/sdav15. . .[b,71,255]/dev/sddx15Accordingly, based on Tables 1 and 2, there are 128 possible disks that are allowed in Linux (i.e. /dev/sda to /dev/sddx), each having up to 15 partitions.
SCSI CD-ROM devices are allocated the block major number of 11. Traditionally “sr” has been the device name, but “scd” probably is more recognizable and is favored by several recent distributions. As shown in Table 3, in Linux, 256 SCSI CD-ROM devices are allowed.
TABLE 3Device Names for SCSI CDROM Devices (scd/sr)[b,11,0]/dev/scd0[or/dev/sr0][b,11,255]/dev/scd255[or/dev/sr255]
SCSI tape devices (st) are allocated the char major number of 9. Up to 32 tape devices are supported in Linux, each of which can be accessed in one of four modes (0, 1, 2 and 3), with or without rewind. Device names for the st devices may be allocated as described in Table 4.
TABLE 4Device Names for SCSI Tape Devices (st)[c,9,0]/dev/st0[tape 0, mode 0, rewind][c,9,1]/dev/st1[tape 1, mode 0, rewind]. . .[c,9,31]/dev/st31[tape 31, mode 0, rewind][c,9,32]/dev/st01[tape 0, mode 1, rewind]. . .[c,9,63]/dev/st311[tape 31, mode 1, rewind][c,9,64]/dev/st0m[tape 0, mode 2, rewind]. . .[c,9,96]/dev/st0a[tape 0, mode 3, rewind]. . .[c,9,127]/dev/st31a[tape 31, mode 3, rewind][c,9,128]/dev/nst0[tape 0, mode 0, no rewind]. . .[c,9,160]/dev/nst01[tape 0, mode 1, no rewind]. . .[c,9,192]/dev/nst0m[tape 0, mode 2, no rewind]. . .[c,9,224]/dev/nst0a[tape 0, mode 3, no rewind]. . .[c,9,255]/dev/nst31a[tape 31, mode 3, no rewind]
The SCSI generic (sg) devices are allocated the char major number of 21. As shown in Table 5, there are 256 possible SCSI generic (sg) devices:
TABLE 5Device Names for SCSI Generic Devices (sg)[c,21,0]/dev/sg0[c,21,1]/dev/sg1. . .[c,21,255]/dev/sg255The SCSI generic device name's use of a trailing letter (e.g. /dev/sgc) is deprecated. Each SCSI disk (but not each partition), each SCSI CD-ROM and each SCSI tape is mapped to an sg device. SCSI devices that don't fit into these three categories (e.g. scanners) also appear as sg devices.
Device names, as embodied by their major and minor numbers, traditional device nodes and the current Linux implementation, have several limitations. Existing major and minor numbers are limited to 8 bits each. This is a limiting factor for some drivers, particularly the SCSI disk device driver, which originally consumed a single major number. Since 4 bits were assigned to the partition index (supporting 15 partitions per disk), this left 4 bits for the disk index. Thus, only 16 disks were supported.
A subsequent change in Linux reserved another seven major numbers for SCSI disks (sd), which has increased the number of supported disks to 128. but large storage arrays, such as RAID arrays, can currently present thousands of logical volumes. However, the current device node implementation stored on normal disk-based-systems in Linux limits the number of devices that can be accessed from user space. For targets (SCSI target devices) that support greater than 128 LUNs we cannot associate all the LUNs greater than 128 with a device node, and hence a physical device file because the Linux SCSI I/O subsystem cannot support more than 128 device nodes at any point in time.
In other words, the number of SCSI devices which may be detected from the user space is restricted to 128 (in the case of disk devices) and to 256 in the case of generic devices because of the current Linux device node implementation. What is needed is a method and arrangement that enables an application, such as a user application, to detect greater than 128 disk devices (sd), or greater than 256 generic devices (sg), so that the application may communicate with disk devices or generic devices in excess of the current limitation, without modifying the existing device drivers or the underlying Linux SCSI subsystem.